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REVIEW OF lacii: methodology 
(A PROJECT EVALUATION OF TECHNICAL ACCEPTABILITY) 


LACIE Project Office 
Lyndon B. Johnson Space Center 


PURPOSE OF LACIE 


LACIE is an experimental program designed to 

1. Evaluate and demonstrate the capability of existing tech- 
nology (remote sensing, data processing and analysis, and 
other associated technologies) to make improved world-wide 
crop production information available to decision makers in 
a cost-effective manner; this test of technology is to be 
conducted in a quasi-operational environment. 

2. Research and develop alternate approaches and techniques 
which, upon evaluation, are qualified to be incorporated 
into the LACIE quasi-operational system where required to 
meet performance goals or to improve efficiency. 

The experiment is structured to have three major elements to meet 
these objectives: a quasi-operational on-line activity to con- 

duct the actual inventory, a reseaich activity to explore alter- 
nate techniques, and a test and evaluation activity to examine 
such techniques prior to their incorporation into the on-line 
system. 

A comprehensive experiment plan was developed to evaluate the 
acceptability of existing technology, to develop new technology 
as required, and to make an acceptable technological approach 
available for a future cost-effective operational system. 

In recognition of the value of periodic technical reviews, LACIE 
schedules formal, in-depth technical reviews by selected techni- 
cal personnel inside and outside LACIE having expertise relevant 
to LACIE technology. These reviews are scheduled at approximate 
6-month intervals, and recommendations are tracked to a logical 
final disposition. 

The primary objective of the LACIE experiment is to assemble, 
operate, and evaluate existing remote sensing technology to pro- 
vide multicountry wheat production information cost effectively 
at the country level outside the United States. 



Specific accuracy performance criteria 1 are stated for at-harvest 
estimates; however, an equally important goal of LACIE is to 
establish how accurately estimates can be made at very early and 
intermediate points in the crop year. 

The three major points of justification for LACIE are the following: 

• Current crop production reporting in foreign areas tends to be 
questionable from the standpoint of accuracy, timeliness early 
season and at-harvest reports late), objectivity, and system 
reliability. 

• Remote sensing technology, not heavily dependent on ground 
data acquisition, appears to offer a cost-effective approach 
to a global crop estimation system that could provide improved 
information to the USDA and to the administration. 

• Previous investigations and analyses indicate the feasibility 
of using remote sensing technology and existing systems to 
support a quasi-operat ional experiment prior to the develop- 
ment of a cost-effective operational system. 

Certain key technical issues require further examination in a 
quas i -operat ional environment in the course of the development 
of an operational system. These include the following: 

1. Can the Land Satellite (Landsat) data and classification 
technology support estimates of wheat acreage in a sample 
segment accurately and reliably enough for useful large-area 
crop acreage surveys? 

2. Are the four Landsat spectral bands adequate to identify 
wheat at appropriate stages of crop maturity? 

3. Does the 0.44-hectare (1.1-acre) spatial resolution of Landsat 
cause excessive errors in estimating the acreage? 

4. Can wheat be identified in the Landsat data without resorting 
to ground-verified field identification? 

5. Can human intervention in the analysis process be made 
repeatable and objective? 

6. Can a sample strategy for acquisition of Landsat data be 
designed to achieve the required accuracies with a manageable 
data load? 


The performance goal for accuracy is to obtain 90-percent accuracy 
for at-harvest estimates 90 percent of the time, that is, 9 years 
out of 10. This is referred to as the 90/90 criterion and applies 
to production, at-harvest, at a country level. 
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7. How can the geographic wheat distribution best be determined 
to properly sample? 

8. Does loss of the segment acquisition due to cloud cover cause 
excessive errors, such as bias? 

9. Is the 18-day Landsat coverage frequent enough? 

10. Are two satellites required? 

11. Can yield models which depend on operationally available 
weather, historic yield, and ancillary data be used to esti- 
mate yields accurately enough for production estimation? 

12. How early can reliable production estimates be made? How 
accurate are these estimates as the season progresses 
through harvest? 

13. Can signature extension be made to work and increase system 
cost effectiveness? 


RATIONALE FOR A THREE-PHASE EXPERIMENT 


LACIE is the first attempt to survey an important crop on a large 
scale at repeated intervals over a wide range of conditions. 
Although the feasibility of the remote sensing technology had 
been established by experiments in confined geographic locales, 
it had not been demonstrated over large areas nor did any analysis 
system exist with the capacity to conduct a true large-area 
inventory. Accordingly, a phased approach with orderly steps 
was chosen. These steps involved expansion in two directions: 
in the technical complexity of the functions performed and in the 
geographic size and difficulty of the area being surveyed. 

The choice of steps in terms of technical functions was dictated 
by the availability of components and the rate at which develop- 
ment of new components was anticipated. The choice of study area 
was based on moving from a known "yardstick" region to a global 
scale in several controlled steps. The phasing was also designed 
to provide some experience with the type of problems expected to 
be encountered in subsequent phases. 

Considering all of the foregoing along with a desire for a rapid 
advancement of the nation's international crop inventory capa- 
bility ar.d the need to utilize an available satellite, a three- 
phase experiment was planned as follows: Phase I was devoted to 

initial experiment design, selection and training of personnel, 
assembly of a quasi-operational analysis system, initial operation 
of the system to estimate crop area, development and preliminary 
testing of yield models, and an initial evaluation of key compo- 
nents of the approach. Phase II is designed to complete the ini- 
tial system as defined prior to Phase I with the incorporation of 
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short-term changes based on Phase I results. In addition, 

Phase II is to expand the testing of the selected methodology to 
a broader geographic scope and for the first time, in addition 
to acreage estimates, to operate all parts of the system required 
to make yield and production estimates on a scheduled basis 
throughout the crop year. Phase III will permit additional 
improvements to the LACIE approrch and includes extensive evalua- 
tion of the LACIE methodology in a quasi -operat ional environment 
over important wheat-producing regions of the world. 

The research, test, and evaluation (RT5E) effort was also planned 
and phased so that improvements to the quasi-operational system 
could be made at an appropriate time and contribute to the devel- 
opment '.nd preliminary evaluation of solutions to key technologi- 
cal problems uncovered. 

A strong accuracy assessment effort is a major part of the experi- 
ment to evaluate the quasi-operational system in terms of estab- 
lished criteria and to uncover possible technological problems 
quickly. Accuracy assessment results guide both the quasi- 
operational activity and the RT5E in terms of delineating areas 
in which changes or improvements are needed. 


SPECIFIC OBJECTIVES OF PHASE I 


LACIE Phase I tasks were defined and scheduled to meet the follow- 
ing objectives: 

1. Select the meat promising technology components to (1) identify 
wheat and estimate its area, (2) estimate yield, and (3) esti- 
mate production. 

2. Complete an overall experiment design (hardware, software, 
sample design) required to support al? three phases. 

3. Implement that part of the analysis system required to esti- 
mate wheat area over most of the hard red wheat region of the 
United States (the Great Plains). 

4. Develop procedures for handling and analyzing large quantities 
of data required in LACIE to meet the planned expansion into 
foreign areas. 

5. Select and train personnel from the three participating 
agencies to implement, operate, and evaluate the LACIE system. 

6. Exercise the system in a quasi-operational manner and estimate 
wheat area over the U.S. hard red wheat region and evaluate 
the results, both against established performance criteria 

for at-harvest estimates and to determine how accurately 
early season estimates can be made. 
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Test selected methods for estimating wheat yield and produc- 
tion prior to implementation of this capability for Phase II. 


8. Conduct parallel and supportive RTf ( E to investigate improved 
approaches . 


9. Conduct initial analyses over selected foreign areas and areas 
in the United States outside the Great Plains yardstick area 
prior to expansion in Phase II. 

10. Develop and implement evaluation plans for subsequent phases 
(II and III). 


11. Implement the additional components of the system required to 
support making quasi -operat ional yield and production esti- 
mates in Phase II. 


RATIONALE FOR AND DISCUSSION OF LAC IE METHODOLOGY 


Selection of technology .— Given the project objectives, with 
specific emphasis on large-area estimation and the relative impor- 
tance of more timely and accurate estimates of foreign production, 
the LACIE design was restricted to use only data that were relia- 
bly available in foreign areas (eg., Landsat imagery, data 
reported by meteorological stations, National Oceanic and Atmos- 
pheric Administration (NOAA) satellite data, and published his- 
torical data for the appropriate political subdivision in the 
selected countries). The LACIE system was designed within a 
framework of constraints originating from several sources. 

Examples of these constraints are the sample segment frequency 
constraints specified by the Goddard Space Flight Center (GSFC) 
and the use of readily available implemented classification tech- 
nology validated by previous studies. The Crop Identification 
Technology Assessment for Remote Sensing (CITARS), the first such 
evaluation for LACIE (ref. 1), was a preliminary test of classi- 
fication procedures proposed for LACIE. Additional considera- 
tions were costs, schedule milestones, available resources for 
system implementation, the specified performance criteria, the 
volume of Landsat data that could be stored and processed, and 
other similar factors. 

Quasi-operational and research elements of LACIE .- The quasi - 
operational system designed was to have the Following characteris- 
tics: (1) to be capable of handling large data loads; (2) to use 

the best available technology that meets the specified performance 
goals and would be cost effective in an operational follow-on as 
opposed to supporting a research and development objective; and 
(3) to produce current estimates of wheat area, yield, and produc- 
tion for selected major wheat-producing regions on a scheduled 
basis throughout the crop season. Changes in the technology 
utilized in the quasi-operational system are made only if the 
current technology does not satisfy the established performance 
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criteria or if an alternative technology is more cost effective. 

The implementation of an alternative technology would occur only 
after validation in an off-line test program. 

A research program was established as a part of LACIE to make 
available alternative technology components in areas where the 
mainline technology was suspect. 

Evaluation rationale .— A significant part of the evaluation of 
the statistical methodology and technical procedures being tested 
in LACIL i> based on a quantitative evaluation of the LACIE survey 
estimates as compared to other reliable independent estimates such 
as SRS estimates for large areas; e.g., states and aggregates of 
states. The suitability of this technology is judged in terms of 
quantitative performance criteria. 

Quantitative performance criteria for LACIE evaluation were chosen 
to represent improved information on world-wide wheat production 
at the country level outside the United States. These criteria 
were stated in terms of (1) accuracy (90/90 at harvest), (2) time- 
liness (acreage and yield estimates based on 14-day-old data), 

(3) continuity (regularly scheduled reporting from emergence 
through harvest), and (4) objectivity (quantification of confi- 
dence of estimates). 

To develop and evaluate the LACIE survey system, the experiment 
was planned to first consider wheat-growing regions of the United 
States where reliable, independent survey estimates and ground 
truth would be available. Several study sites exist in Canada 
to augment the ground-truth data set. 

LACIE survey estimate accuracy is quantified at national and, 
importantly, at subregion levels in the yardstick country. The 
subregional analysis is conducted to determine the country depend- 
ent survey performance in foreign areas with analogous agricul- 
tural/climatic conditions. 

Crop survey estimates from established reporting agencies (the 
Bureau of Census and the SRS) in the yardstick country were 
assumed to be accurate enough for evaluation of LACIE estimates. 
Phase I experience indicates this to be a reasonable assumption 
at the national level. However, at the state level and below in 
the United States, SRS estimates were not sufficiently accurate 
for all purposes. Additional ground truth was acquired in Phase I, 
and the s.cquisition of more ground truth is planned in Phase II. 

A statistical experimental design was developed to estimate the 
magnitudes of system subcomponent errors, such as classification 
and sample error (both bias and random components), analyst and 
machine error, yield estimation error, and production estimation 
error, so that the technology can be intelligently improved as 
required. 

In addition, a system simulation/error model has been developed 
to estimate the effects on accuracy of various sample designs, 
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cloud cover effects, and different agricultural/climatic condi- 
tions. This simulator is being used to investigate the accuracies 
obtainable with the LACIE technology in various regions and with 
various proposed changes to the technology. 

In Phase I, major emphasis was devoted to identifying significant 
problem areas and incorporating rjcessary changes into the on-line 
system. A final evaluation of these improvements was made late in 
Phase ? on the basis of a final at-harvest analysis of all data. 
The evaluation of the early season estimates for Phase I could not 
have been completed much before an assessment of the accuracy of 
the early season Phase II estimates. Therefore, a decision was 
made to concentrate the LACIE efforts at that time on the real- 
time evaluation of Phase II early season estimates. Thus, major 
attention is being given to establishing the accuracy with which 
early season estimates can be made in Phase II. 


REVIEW OF LACIE ACREAGE ESTIMATING METHODOLOGY 


Sampling frame and sampling design .— Based on known constraints, 
a total allocation of 4800 sample segments was divided among the 
selected countries using the criteria and procedures in refer- 
ence 2. (Sample segment location constraints are also specified 
in tliis document.) The sampling strategy selected for use in the 
United States was designed to utilize county-level statistics for 
segment allocation. This strategy was implemented based on the 
following rationale. 

1. Such a sample design would provide the most accurate estimate 
in the United States, the selected "yardstick" area, for which 
historic statistics were available down to the county level to 
support missing data resulting from cloud cover. By utilizing 
the most accurate available sampling strategy in the United 
States, the impact of having less accurate data in foreign 
countries could be determined throigh use of a "degraded" data 
set obtained by employing various subsets of the U.S. data. 

2. Statistics normally produced by the USDA would be adequate to 
evaluate sampling error without the need for extensive ground 
truth . 

3. The methodology for using Landsat imagery and agrophysical 
data was not sufficiently developed at the start of Phase I 
to select strata along natural boundaries; hence, strata 
would need to be defined by political boundaries. 

4. The implemented strategy would provide data of sufficient 
quantity and quality to meet required performance levels and 
also satisfy the existing constraints. 

Throughout the life of the experiment, attention will be devoted 
to other key sampling- related issues such as optimal sample seg- 
ment size, the optimal number of sample segments, and potential 


7 



relaxation of engineering constraints on sample segment location. 
The exact formulations used in the sample design, expansion (or 
aggregation) process are shown in reference 2. 

In the LACIE framework, the substrata are identified as Group I, 
Group II, or Group III (county- level designations in the United 
States). These designations are made on the basis of •» threshold 
value defined in reference 2. However, the qualitative defini- 
tions are as follows: 

Group I substrata — intensive wheat -produc ing areas which are 

allocated one or more sample segments. 

Group II substrata — areas which produce some wheat; sample seg- 
ments allocated based on PPS; a sample seg- 
ment represents more than one substrata. 

Group III substrata — areas historically having very little wheat; 

thus, no sample segments are allocated. 

The basic rationale underlying tli. scheme is that such an allo- 
cation scheme appeared to produce the most efficient utilization 
of the data available for stratification and would give better 
segment coverage to major producing areas and would thus improve 
the probability of an accurate estimate. 

Although certain engineering constraints were considered in the 
implementation of this general approach, the evidence examined in 
Phase I indicated that these factors did not significantly 
impact accuracy. A majority of these constraints are no longer 
inherent in the system, and an improved Phase III sample design 
will not be limited by such constraints. 

Phase I experience indicated that the design discussed above had 
two main disadvantages: (1) the approach may not be optimum to 

indicate the expected performance levels in foreign regions; that 
is, the U.S. county is a substrata of much smaller size than the 
areas for which data wore available in most foreign regions, 
and (2) sophisticated methodologies are necessary to evaluate 
the precision of area, yield, and production estimates. Although 
the methodology utilized is correct, considerable effort is 
required to establish the degree to which all assumptions are 
sufficiently satisfied; and extensive data are required to conduct 
the evaluation of precision. 

Although the LACIE selected sample segments are probably larger 
in size than might be optimum for a future operational system, the 
planned analysis of within- segment and between- segment variation 
in LACIE Phases I and II will permit selection of a more optimum 
segment size and total segment number for Phase III in LACIE. 
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CROP CLASS! FICA : iON AND AREA ESTIMATION METHODOLOGY 


The rationale on which the choice of the classification and per- 
segment area estimation technology was based was to select (1) the 
best available tested and implemented technology; (2) technology 
to identify wheat in the Landsat data without the use of ground 
truth; (3) the most efficient repeatable analyst procedures to 
process the large data Quantities; (4) procedures to satisfy a 
need for timely estimates and the resultant requirement for rapid 
analysis of a segment; and (5) procedures to minimize human inter- 
vention required to assure cost effectiveness. 

Much work had been done to establish a remote sensing technology 
base extending from 1966 when computerized pattern recogni- 
tion analysis techniques had been coupled with electromagnetic 
energy measurements to classify major agricultural crops, through 
1973 using Landsat-1. These results provided the basis on which 
to select the LACIE crop classification and estimation technology 
and were promising enough to have established the feasibility of 
conducting gross inventories of the areal extent cf agricultural 
crops . 

As pointed out in "Purpose of LACIE," certain key technical issues 
required further examination in a quasi -operat iona 1 environment. 
These issues motivated the design and activation of an extensive 
accuracy assessment effort, specifying ground-truth data collection 
and comparative analysis techniques to aid in the resolution of 
these issues. 

Classification and proportion estima t ion.- 
concern in any estimation problem is bias . 

LACIE area estimation system arc known to 
or overestimates of the area. 

Any estimation system must make provisions to establish the degree 
to which the system estimates are biased and to determine whether 
or not the magnitude of the bias is acceptable in terms of some 
success criterion. The coefficient of variation (c.v.) is used 
in LACIE as conventionally defined and includes variation due to 
classification and sample random errors. LACIE also includes an 
estimate of the bias to account for total mean-square error. 

In addition, the LACIE experiment design includes methods for 
indicating the magnitudes of the sources of the bias and variance 
of the area estimates, such as the training set size, the time of 
year or the growth stage at which wheat area was estimated, the 
effect of the analyst, the effect of the classifier, the effect 
of Landsat spatial resolution, and other possible error sources. 

This testing indicated that classification procedures employed 
initially in the quasi-operational system did indeed have a sig- 
nificant source of bias. However, these test results allowed 
the source of the bias to be isolated and at least partially 


A source of major 
Many factors in the 
result in underestimates 
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corrected in Phase I. The bias resulting from current procedures 
appears to be acceptably small for both Group 1 and Group II seg- 
ment area estimates. 

In order to improve the cost effectiveness of the LACIE system, 
signature extension techniques are being evaluated in Phase II 
and will be incorporated it found effective. 

The effect of the human analyst on the system will be continually 
monitored although tests to date indicate that documented data 
analysis procedures, developed to reduce analyst subjectivity, 
have been effective in addressing this potential problem. 

Finally, an expanded probability ground data set will be collected 
in Phase II to evaluate potential error sources discussed above. 

As problems are uncovered by these assessments of the quasi- 
operational system, requirements for alternate technology are 
being defined for the LACIE RTf ( E program whose effort is focused 
on developing and evaluating solutions to key technological 
problems. The products of this effort, if validated by the LACIE 
test and evaluation program, will either be incorporated into the 
quasi-operational system or stand alone as potential solutions at 
the end of Phase III. 

Nonrespons e ct to cloud cover .— LACIE does not get coverage over 
every segment of every pass because of atmospheric effects, such 
as haze and cloud cover. To counteract this problem, a sample 
strategy was designed that, in principle, will minimize the effect 
of such loss. However, the magnitude of this effect is being 
monitored in the accuracy assessment program, and it is planned 
to continue to look for significant effects due to missing data 
over the full range of geographic conditions in which a future 
operational system needs to perform. 


TIMELINESS 


The timeliness goal established for the LACIE area estimation 
system is to process the Landsat data within 14 days after acqui- 
sition when projected into an operational environment. Since 
LACIE is basically a one-shift, 5-day-a-week operation, the 
processing time goal is 29 days, allowing for queuing times 
in the one-shift operation. However, this time could be rea- 
sonably expected to reach the 14-day goal in the anticipated 
operational environment of three shifts, 7 days per week. 

Yield estimates are based on monthly averages and are generated 
about 4 days after the end of the month. Production estimates 
are made about 10 days after the end of the month 

LACIE is faced with a data timing difference in estimates which 
complicates accurate comparisons. For example, LACIE area is 
based on emerged wheat which should, through the growing season, 
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converge to area available for harvest. The SRS area estimates 
are based primarily on planted area until June. 


REVIEW OF YIELD FORECAST METHODOLOGY, INCLUDING MODELING, 
INPUT DATA, AND FORECAST PRECISION 


Yield model.— LACIE initially selected a regression model devel- 
oped by NOAA to test and evaluate in Phase I prior to quasi- 
operational testing in Phase II. 

These models were run over different size geographic regions to 
determine the proper size of regions such that when yield is merged 
with area to estimate production and aggregated to the country 
level, LACIE accuracy performance criteria would be met. 

The Phase I analyses of the model incorporated into the qua«i- 
operation in Phase II indicated that at-harvest estimates of 
yield coul< be expected to meet the LACIE 90/90 accuracy crite- 
rion. Early season estimates also appeared to be quite good 
over the 1974-1975 season. The Phase I tests consisted of run- 
ning the models in real time during the 1974-1975 crop year and 
with historical data each year for a 10-year period prior to 
that . 

It is recognized, however, that the models may not perform as 
well in foreign areas where historical record data are lacking 
or nonexistent. For this reason, LACIE is expending considerable 
effort in the RTtiE program to develop improved yield estimation 
models. These efforts are concentrating on approaches that model 
yield in terms of variables that are closely related to the plant 
growth processes and that are significant determinants of yield. 

Research tasks have also been initiated to develop improved 
methodology for estimating the trend factor(s) for the LACIE 
yield models. Sensitivity analyses in Phase I clearly indicated 
"the importance of properly estimating trend. 

In Phase I, LACIE evaluated the best candidate models developed 
by the Earth Satellite Corporation (Earthsat) and initiated 
development of yet another model at Kansas State University 
(KSU) . The LACIE project made the results of the Earthsat model 
runs available to the SRS, which is currently evaluating the 
approach for applicability. 

Use of variables (input data) .— The input variables used in the 
LACIE model were selected on the basis of statistical analyses 
which established a significantly strong correlation between 
average monthly values of the variable and final yield. Specifi- 
cally, it was found that the major factors or variables are tem- 
perature and precipitation. LACIE is currently evaluating yield 
models that make use of weekly or daily averages of these variables. 
In addition, the LACIE is researching improved approaches to esti- 
mating variables found to be important indicators of yield. 
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SIGNIFICANT RESULTS OF PHASE 


▼ 


Quasi-operational acreage test.— After correction of significant 
implementation problems in the initial quasi-operational area 
estimation system, the resulting wheat area estimation at harvest, 
based on its performance quantified over the U.S. Groat Plains, 
was deemed marginally satisfactory in consideration of the 90/90 
at-harvest criterion ^or wheat production estimation. The cri- 
terion can be met by a reasonable increase in the number of sample 
segments acquired and/or with improved stratification of agricul- 
tural and nonagricultural areas in selected regions. The area 
estimation system shows a tendency to underestimate when compared 
to the SRS estimates; that is, statistical analyses show a sta- 
tistically significant difference between the two estimates. A 
significant contribution to this is believed to be a sampling 
problem in North Dakota. 

The LACIE Great Plains area estimate was approximately 18 million 
hectares (46 million acres) compared to the SRS estimate of approxi- 
mately 20 million hectares (51 million acres), or about 10 percent 
below the SRS figure. This represents a statistically s nificant 
difference from the official USDA December 1975 year-end estimate of 
acreage for the yardstick region. The major source of this bias is 
due to the sample allocation in North Dakota. An improved allocation 
of samples on the basis of a better partitioning of agricultural lands 
into more homogeneous strata is expected to reduce any bias to a tol- 
erable level. The use of full-frame Landsat imagery is critical to 
defining adequate strata to avoid such sampling error; this improved 
sample allocation is curiently planned to be tested in Phase III. 

The c.v. computed for the LACIE acreage estimator, when projected 
to the U.S. national level, is about 4.5 percent, a fraction of a 
percentage point above the 4.25 percent required if production esti- 
mates are to meet the 90/90 criterion. Since data loss due to cloud 
cover and early implementation problems resulted in a reduction in 
the number of LACIE sample segments used (of 411 allocated, 272 were 
satisfactorily acquired, processed, and aggregated), this random 
error component can very likely be reduced to its acceptable limit 
of 4.25 percent or below by the improvements implemented and planned 
for Phases II and III. 


The results of this quasi-operational te 
examined in the Phase I production feasi 
LACIE area estimates were combined with 
the resulting production estimates were 
tior. estimate satisfied the 90/90 criter 
basic compatibility of the LACIE area an 
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LACIE accuracy assessment methodology will be summarized in the 

final LACIE Phase I Evaluation Report to be released shortly. 
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Results indicated that the Landsat data and the classification 
technology can estimate the small grains (i.e., wheat and closely 
associated small grains) area within a sample segment accurately 
and reliably enough to meet the LACIE goals. Overall, the LACIF. 
estimates in a 9- by 11-kilometer (5- by 6- nautical-mile) 
segment agree well with ground and aircraft determined area 
within these segments. In North Dakota where 20 such sites 
were examined, no significant difference was detc^tei between 
the LACIE and ground-truth estimates over the 9 11-kilometer 

(5- by 6-nautical-mile) segments. The estimated c.v. of the 
random classification error was "acceptably" small. These 
analyses confirmed that bias introduced by various factors, such 
as Landsat spatial resolution, lack of spectral resolution, 
classifier (analyst - interpreter) bias and repeatability, is not 
excessive in terms of the required performance criterion. 

Results of these tests did indicate a difficulty in differen- 
tiating wheat from other closely related small grains. However, 
satisfactory wheat area estimates were obtained through the reduc- 
tion of the small-grain area estimates in accordance with relative 
amounts of these crops as determined from historic data; and these 
procedures are being refined. 

The accuracy assessment program also indicated that an incorrect 
classification procedure initially implemented in the quasi- 
operational system did introduce a significant bias into the area 
estimate; however, these test results allowed the source of this 
bias to be isolated and corrected. 

Other detailed tests indicated that small-grains identification 
can be adequately established, through the use of human analysts, 
directly from the Landsat data without ground truth. The repeata- 
bility of the analyst performance was also verified in duplicate 
analyses of a common set of segments by several analyst teams. 

There are some indications that in regions with marginal wheat 
production with small fields or large amounts of confusion crops, 
wheat identification may be more difficult than in more intense 
producing areas. LACIE plans to monitor these situations closely 
during Phases II and III. 

Fundamental issues involving sample design were also partially 
answered by the Phase I accuracy assessment. The several approaches 
taken to estimate sample error indicate that for the U.S. Great 
Plains, except North Dakota, it is acceptably small given all the 
allocated segments. Loss of acquisitions from cloud cover was a 
problem in Phase I; however, tests conducted to date indicate that 
error arising from this loss is probably random in nature with no 
significant bias being introduced. 

In North Dakota, a significant underestimate of the wheat area was 
observed. Further analyses indicate the major problem is with the 
sample placement as opposed to the classification analysis. Indi- 
cated solutions are the allocation of additional samples or 
improved stratification to reduce agricultural area variability, 
or both. 
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Yield feasibility.— The Phase I analyses of the model to be incor- 
porated into the LACIE quasi-opcrational system in Phase II indi- 
cated that the Phase II model can be expected to support the 90/90 
criterion in regions having characteristics similar to the geography 
and agriculture represented by each of a majority of the states in 
the yardstick region. It is recognized, however, that the models 
may not perform as well in foreign areas where historical record 
data are lacking or nonexistent. 

In a test of the yield models over the years 1965 to 1975, the c.v. 
of the yield estimates was on the order of 2 percent at the national 
level, lower than the 4.25 percent required. However, it was noticed 
that in 5 of the individual years, the difference in the average 
yield predicted by LACIE and the SRS was large enough to indicate a 
statistically significant difference for those years, even though 
over all test years the LACIE overestimated about as frequently as 
it underestimated SRS. A more conservative test was devised in 
which, for these same test years, the LACIE at-harvest yield esti- 
mates for each year were combined with the SRS area estimates for 
that year to produce production estimates referred to as LACIE test 
production estimates (TPE). The difference between the LACIE TPE 
and the SRS production estimates were then computed for each test 
year. Since for each year SRS a-ea and SRS yield figures combine 
to precisely the SRS production estimate, the differences between 
the LACIE TPE and the SRS production estimates could be attributed 
solely to the area-weighted differences between the SRS and LACIE 
yield estimates. A hypothesis test was developed which showed that 
in case the differences between the LACIE Great Plains TPE and the 
SRS Great Plains production estimates were within ±9.5 percent of 
the SRS estimate in 8 of 10 years, then the LACIE yield estimator 
could be expected to meet the at-harvest country level 90/90 cri- 
terion. This criterion would be met when the yield estimate is 
combined operationally with an unbiased area estimator in which 
the errors are uncorrelated with the yield estimator errors and 
for which the random error component is equal to the random error 
component of the yield estimator. This test indicated that the 
LACIE yield estimator would meet only about an 88/90 criterion. 
However, this test and others indicated that one source of error 
in the yield models was the form of the models which resulted in 
unrealistically high or low yields for extremely high or low 
values of the temperature or precipitation. An improved model 
has been developed. Tests of this improved model indicate that 
it will significantly improve estimates. 

Production feasibility test .— When the LACIE area estimates and 
the LACIE yield estimates are combined, the resulting production 
estimates satisfy the 90/90 criterion. In the Great Plains, the 
LACIE production estimate was 8.8 percent below the SRS final 
estimate for the same region. The c.v. of the LACIE production 
estimate was 5.3 percent at the Great Plains level and 4.2 percent 
when projected to the national level. This is within the accept- 
able tolerance of 6 percent for an unbiased estimator. Since 
the difference between the SRS and LACIE estimate at the Great 
Plains level is not significant (i.e., could likely be a random 
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fluctuation in this statistical quantity), the estimator can be 
judged to satisfy the 90/90 criterion since the c.v. is less than 
the 6 percent required. The largest regional problem observed is 
once again in North Dakota where production is significantly under 
estimated because of the area underestimate discussed earlier. 


CHANGES IMPLEMENTED AND PLANNED FOR PHASES II AND III ON THE 
BASIS OF LAC IE EVALUATION OF PHASE I 
TECHNOLOGY AND APPROACH 


Early in LACIE, areas of technical risk were identified, and 
research and development efforts were initiated to provide for 
future system upgrades. The results of Phase I were also tracked, 
and in August and September 1975 project reviews were held with 
technical personnel inside and outside LACIE with expertise rele- 
vant to LACIE. These reviews aided in solidifying areas of needed 
improvement, and these improvements were made to the analysis sys- 
tem. In March 1976, a further project review was held to review 
the implemented and planned improvements, and additional recommen- 
dations were formulated for system improvements. On the basis of 
this review and as improvements are formulated and implemented 
into the system, additional reviews will be held. The most sig- 
nificant technology changes resulting from these project actions 
are the following. 


Technology Changes Implemented for Phase II 


Team approach .— Early design consideration involving signature 
extension and limitation on the time available to hire and train 
analysts resulted in an approach of having separate image inter- 
preters and data proctssing specialists sequentially involved in 
the analysis of a single segment. This approach did not allow 
the analysts sufficient visibility and understanding of cause- 
and-effect relationships to establish quality control over results 
As a result, for Phase II, analysts were provided additional train 
ing and now work in teams. The approach has demonstrated benefits 
in accuracy, efficiency, and reduced cost overhead. 

Improved acquisition plan and analysis concept .— At the beginning 
of Phase I, the LACIE Fall estimates of winter wheat were signifi- 
cantly higher than the SRS estimates. LACIE correctly classified 
bare soil from the Landsat data but improperly summed these with 
the proportion estimates. This problem was compounded by the fact 
that early biowindow Landsat data were acquired too early and that 
in many fields wheat which had been planted had not yet sprouted. 
Since then, crop growth stage models have been incorporated in the 
LACIE operation, and Landsat data can be selected at the proper 
time. 
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Crop calendar starter models .- The LACIE analyst requires crop 
calendars which define the stage of growth of the plant in order 
to discriminate wheat (or small grains) from other ground cover. 
Adjustment of the normal crop calendar is required to account 
fcr the current year change due to climatic variation. During 
Phase I of LACIE, the adjusted crop calendar model, based on 
Robertson's biometeorological time scale, performed satisfacto- 
rily in the Great Plains when the model was initiated with actual 
planting dates for spring wheat and jointing dates for winter 
wheat. Realizing that actual dates will not be available for 
foreign regions, concentrated efforts were undertaken to develop 
both winter and spring wheat starter models. This resulted in a 
winter wheat end-of -dormancy restart model and a spring wheat 
planting date starter model being implemented for LACIE in the 
spring of 1976. 

Accuracy assessment .— The LACIE Phase I Accuracy Assessment 
activity in the United States concentrated on the analysis of the 
intensive test site (ITS) data. In order to cover wider geographic 
areas and better assess the LACIH operations, some regular LACIE 
segments in Montana and North Dakota, the so-called "blind sites," 
were ground truthed late in the Phase I operation. These were so 
successful at giving rapid and accurate feedback into the system 
that the number of such segments in the U.S. Great Plains has 
been increased for Phase II, and this effort is now oriented to 
provide data for early season as well as at-harvest evaluations 
of LACIE performance. 

These data are to be used to evaluate classification error, not 
sampling error; and in order to be cost effective, it is neces- 
sary to randomly select the segment after a large number of acqui- 
sitions have occurred. This selection process will tend to weight 
selection toward regions which have lower cloud cover probabilities. 
However, for this procedure not to be biased, it is only required 
that the cloud cover probability be uniform over a county. In order 
to augment the sampling accuracy, critical data from the SRS 1.6- by 
1.6-kilometer (1- by 1-mile) segments are being provided for the 
1973-1974 and 1974-1975 crop years. 


In Canada, for Phase II, the accuracy assessment activity will 
focus on 10 ITS's. Data from these sites will support investiga- 
tions into yield, crop calendar accuracy, and an effort similar 
to the one carried out in the United States to assess classifica- 
tion accuracy with respect to training set size, biostage, and 
analyst teams. 

In the U.S.S.R., previous years' Landsat data will be ordered and 
analyzed by LACIE for several key indicator oblasts. Classifica- 
tion accuracies from ITS's in Canada and the United States will 
be projected in the U.S.S.R. country aggregation by use of the 
LACIE simulation models. 

Sampling accuracy will be determined by image interpretation of 
full-frame imagery in the United States and in foreign areas. 
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Monitoring "standard statistics" based on the LACIE estimates 
will also be used to infer how overall accuracy changes from 
planting to harvest in the United States and foreign areas alike. 

Foreign exploratory segments. — Exploratory segments in foreign 
areas and in regions of the United States outside the Great Plains 
were defined to gain early indications of technical feasibility 
before expansion in foreign regions. A review of Phase I results 
produced new recommendations for Phase II exploratory segments. 
Segments were reselected in several countries to make the explora- 
tory analyses more meaningful, either through inclusion of total 
political subdivisions or through better representation of the 
agriculture. Secondly, new analysis techniques are being evaluated 
for the small field problem identified in China and India. 


Technology Changes Planned for Phase III 


Upgraded yield models .— In addition to providing acceptable 
yield estimates, the Phase I test of yield models and the parallel 
supporting research emphasized the need for the planned specific 
model changes needed for a more responsive system. Key limita- 
tions to existing models are failure to account to a sufficient 
degree for climatic effects on yield as a function of crop devel- 
opment stage, refinement of model meteorological input to a daily 
basis, and dependence on historical data records of varying 
quality which may not be consistent with homogeneous yield strata. 
Two improved model forms are under development, one of which will 
be selected for implementation in the yardstick region and in the 
U.S.S.R. spring wheat indicator region early in Phase III. 

The modeling approach at KSU is tc derive yield response to the 
environment from historical plot data, determining partial deriva- 
tives that describe the reduction from an area yield potential 
caused by environmental effects. This form provides a great deal 
of flexibility of choice in the input weather-related variables 
as might be required in data sparse regions. It is very adaptable 
to stratification of areas of homogeneous yield potential and pro- 
vides a definite step toward a general model. This model form is 
expected to be an improvement over existing models in areas with 
little or no historical data. Further, it can emphasize and place 
reliance on additional point meteorological data. This model can 
also accommodate remotely acquired data. 

The improved NOAA/Center for Climatic Environmental Assessment 
(CCEA) model form is more area-specific in the applicability of 
its coefficients than is the KSU model; however, it is anticipated 
that it will account for seasonal weather anomalies (by incorpora- 
tion of crop calendar into yield model). A soil moistur • budget 
will replace precipitation as an input to account for run-off, 
and an improved definition will be made of regions where plant 
response to weather is uniform (stratification). These models 
are to be available for quasi-operational testing by April 1977. 
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Revised sampling in key areas .— A methodology has been developed 
to define sample strata along meaningful boundaries of homogeneous 
agricultural areas defined by contours of equal agricultural 
density and variability as determined from Landsat imagery and 
by soil and climatic relationships. Activities are under wuy 
to implement this early in Phase II for the yardstick region in 
the United States and the U.S.S.R. spring wheat indicator region. 

In addition, it is believed that this change will provide a 
sampling plan that offers more insight into the performance to 
be expected in foreign areas. 

Signature extension .— The objective of signature extension is to 
allow an increase in sample segments analyzed without substan- 
tially increasing analysis costs in an operation. This was docu- 
mented as an area of high technical risk in the initial days of 
LACIE. The first attempts in Phase I were, for the most part, 
costly and unsuccessful. The supporting research and development 
element of LACIE indicated that improved technology is available 
in the areas of stratification to define segments of similar spec- 
tral characteristics and in the area algorithms for correction 
of atmospheric variations. This technology is being evaluated 
under test conditions and, if successful, can be transferred to a 
quas i -operat ional test in late Phase II. The quasi-operational 
test will provide the basis for a decision to implement signature 
extension into the operation in Phase III. There is no technical 
reason to believe that signature extension capability cannot be 
developed . 

Clustering .- Clustering is a technique that has been widely used 
for the analysis of mult ispectral data. It was strongly recom- 
mended as a tool for the analyst to use in defining the spectral 
structure in a scene. The results of Phase I indicate that it 
would be especially valuable to the analyst in attempting to per- 
form multitemporal analysis when the analyst has several images 
to contend with. Implementation in Phase III is contingent on 
successful development and test efforts now under way. 

Revised thresholding .— Thresholding is used in segment analyses 
to extract extraneous picture elements (pixels) from the propor- 
tion estimate; that is, pixels which contain little or no infor- 
mation related to wheat (e.g., data dropouts, clouds, and shadows). 
In addition, it is useful as a diagnostic tool in the evaluation 
process; for example, in the detection of missing training signa- 
tures. In Phase I, *:he threshold model resulted in undesirable 
levels of thresholding, and the thresholded pixels were not being 
treated correctly in the proportion estimate. Improvements are 
being incorporated in the threshold model for Phase III. Test 
show that this will yield improved estimates relative to the 
former model, specifically when multitemporal classification 
is utilized. 

Analyst-interpreter keys .— During LACIE Phase III, the analysis 
of Landsat data will employ newly developed ana lyst - interpreter 
keys which group spectrally similar segments and provide in a 
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single document the ancillary data and decision logic for deter- 
mining crop identification within each group of segments. The 
keys will provide interpretative methodology and analytical 
tools for improved wheat and/or small grains identification from 
the earliest crop development period. 


GENERAL OBSERVATION OF EARLY SEASON PHASE II RESULTS 


Evaluations of the LACIE U.S. Great Plains area, yield, and pro- 
duction estimates in Phase II by the LACIE accuracy assessment and 
Information Evaluation staff (USDA LACIE staff in Washington) have 
found that the LACIE estimates agree well with the SRS estimates. 
Comparisons of LACIE area estimates to the Phase II blind site 
estimates indicate that the LACIE estimate of standing and vigorous 
wheat area is also in good agreement with actual ground conditions 
as determined from aircraft color infrared (CIR) imagery and ground 
observations and thus reflect wheat area available for harvest 
quite accurately. In the current crop year, late planting, drought 
conditions, green bug infestation, late grazing, and winter kill 
are affecting significant amounts of wheat, causing stands which 
are not recognizable from Landsat or aircraft CIR imagery. 

To date, LACIE has released for the U.S. Great Plains five monthly 
reports on wheat area, yield, and production. These reports are 
delivered to the USDA LACIE staff in Washington, D.C., prior to 
the release of the SRS reports. The Information Evaluation staff 
in Washington compares the LACIE estimates to the SRS estimates 
after the latter are published. 

A problem exists in evaluating the LACIE early season yield and 
production estimates in that no technique other than the LACIE type 
yield models exists for making objective early season yield esti- 
mates. In the yardstick region, the SRS does not normally make 
early season wheat production estimates before May, other than 
by trend. In April 1976, the SRS resorted to a weather model 
very similar to that used by LACIE to make early estimates of 
wheat yield in April. The April SRS estimate was made as the 
result of a special request from the U.S. Secretary of Agricul- 
ture because of drought conditions. 

For the 2 months thus far in Phase II (April and May), when side- 
by-side comparisons are possible, LACIE yield results appear to 
be tracking reasonably well with the official SRS estimates. In 
April, the LACIE yield estimate in the Great Plains was slightly 
higher than the SRS estimate. In May, the SRS yield estimate 
increased and the LACIE yield estimates for the same period 
decreased and both estimates now agree well. 

The first LACIE estimate of area, yield, and production in a 
foreign country was accomplished in Phase II using fall collected 
data over an important winter wheat (indicator) region of the 
U.S.S.R. The LACIE estimates are lower than those for a normal 
year, but the trend of this reduction is in line with the current 
weather situation in that region. 
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ECONOMIC EVALUATION AND TRANSFERABILITY APPROACH 


The plan for economic evaluation of LACIE is comprised of two 
parts, (1) the cost of an operational wheat estimation system 
within the USDA and (2) the value of foreign wheat production 
estimation co USDA and Administration decision-makers. This 
cost is expressed in terms of men, monies, and materials as pro- 
jected into an operational environment as opposed to NASA, NOAA, 
and USDA resources required in the LACIE environment, including 
RT6E and the experimental quasi-operat ional system. The planning 
guidelines and facts that underlie the project approach are based 
on experience gained in LACIE and in addition draw heavily upon 
operations research techniques. The following characteristics 
of LACIE need to be understood. 

1. The physical system (hardware, software, and procedures) 
supporting LACIE at JSC, GSFC, and CCEA is comprised of 
diverse pieces of existing systems, originally designed 
for somewhat different applications than LACIE. Thus, 
they are experimental in nature and are not representative 

of a facility designed specifically for a production-oriented 
wheat inventory task. 

2. LACIE is an experiment being supported by activities over and 
above the quas i -operat ion . Only about one-third of the 
assigned personnel are dedicated to the operations portion 

of the experiment, the majority being applied to general 
experimental support tasks. 


A USDA operational wheat estimation system is being cost-simulated 
on the basis of task specific, state-of-the-art hardware, soft- 
ware, and procedures. This cost factor will be evaluated in a 
manner similar to that used by private industry and the U.S. 
Department of Defense; for example, an acceptable cost for the 
production system would be a cost rang that falls within the 
ratio of 5 to 1, where 5 equals system development cost and 1 
equals any 1 year's production cost. 

The benefits accrued from a production system are derived from 
the impact of geographically broader, more timely, accurate, and 
objective wheat information available to the decision-makers. 
Benefit areas are to be identified through a plan to interview- 
decision-makers at various levels throughout USDA to assess worth 
in terms of (1) the types of decisions made; (2) the use of crop 
information and its value to respondents; and (3) the potential 
effect on USDA's responsibilities including USDA policy decisions, 
program actions, publication of reports to the agricultural commu- 
nity, ability to analyze and respond to changes in policy, and 
improvement in forecasts having economic consequences. In addi- 
tion, modeling techniques will be used to assess the impact of 
improved wheat information on commodity price forecasting efforts, 
farm income, target price and load levels, government costs under 
alternative programs and decisions, and consumer food decisions. 
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An integral part of the project plan, the output of this review 
will he applied, together with the technology and procedures tht . 
are being developed within the experiment, to the design of a sys- 
tem that could be placed directly into production at a USDA 
facility. 


REFERENCES 


1. Bizzell, R. M. ; Feiveson, A. H.; and Hall, F. G. : Crop 

Identification Technology Assessment for Remote Sensing 
(CITARS). Vol. X, Interpretation of Results. JSC-09393, 
Dec. 1975. 


2. Large Area Crop Inventory Experiment (LACIE) . Vol. IV, 
Cro ^ Assessment Subsystem (CAS) Requirements. Rev. A. 
LACIE-C00200 (JSC-10808), Dec. 1975. 


21 



GLOSSARY 


CITARS - Crop Identification Technology Assessment for Remote 
Sensing; a project to evaluate the effectiveness of various 
methods of analyzing remotely sensed agricultural Jata (con- 
sidering various crops, atmospheric and seasonal factors, and 
sensors) conducted by NASA/JSC in cooperation with supporting 
universities and federal agencies. 

olaoaif ioation — as applied to remote sensing, a semiautomated, 
computer- implemented process in which subsets of measured values 
of electromagnetic energy reflected from ground cover types within 
a geographic area (training area) are uniquely associated with the 
prevailing cover types in that area. Measurement values similarly 
obtained from cover types outside the training area are then labeled 
or classified according to the associations developed within the 
training area. 

cluetering — mathematical procedure for organizing mu 1 1 ispect ra 1 
data into spectrally homogeneous groups. 

crop calendar — calendar depicting the period of growth, develop- 
ment, or biological stages of the major crap types within a speci- 
fied region during a calendar year. 

data processing - application of procedures (mechanical, elec- 
trical, computational) by which data are changed from one form 
to another. 

ground truth — all the information known about a selected location 
resulting from records, in-situ measurements, and field observa- 
tions . 

ITS — intensive test site; area selected as a calibration standard 
for remote sensing operations. Ground truth information is col- 
lected in conjunction with each set of remotely sensed data 
acquired . 

LACIE — Large Area Crop Inventory Experiment; joint experiment 
of the U.S. Department of Agriculture, the National Oceanic and 
Atmospheric Administration, and the National Aeronautics and 
Space Administration to prove out an economically important 
application of remote sensing from space. A series of experi- 
mental investigations utilizing multispectral and meteorological 
data to identify and measure the real extent of major crop types 
and to estimate their yields. 

Landsat — Land Satellite; designation for two NASA earth resources 
remote sensing satellites equipped with multispectral scanners and 
return beam vidicons, the first (Landsat-1) of which was launched 
1972 and the second (Landsat-2) in January 1975. 
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pixel — picture element; dual record of a single data-takc during 
the scan of an electro-optical scanning sensor and referenced by 
scan line and column. 

PPS - probability proportional to size; one-stage variable proba- 
bility sampling applied in many types of surveys (agriculture, 
forest, and census). The probabilities of selection are defined 
as 
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in which is a prediction, obtained by remote sensing of the 
resource quantity in the ith population unit. 

real time — term used in electronic data processing to describe 
operations conducted at the same tine as, or concurrently with, 
the collection of input data. 

remote sensing - discipline concerned with conducting aerial or 
space surveys of the earth's surface using all types of sensing 
devices; space, airborne, or other data collection platforms; 
manual and automated data processing equipment; information 
theory and processing methodology; information theory and 
devices; and large- sys terns theory. 

remote sensing technology — technology based on measurements of 
electromagnetic energy reflected or emitted from scenes or objects. 
The five elements of remote sensing technology are data acquisi- 
tion, data correction, data analysis, information management, and 
appl icat ions . 

sampling — statistical procedure using portions of a subject of 
interest to represent the properties of that subject. 

signature — color, tone, brightness, texture, and pattern of a 
scene or object; the overall visual characteristics of a crop as 
it appears on multispcctral imagery. 

signature extension — extrapolation of a known set of remotely 
sensed data to classify other remotely sensed data. 

signature extention accuracy - measure of the accuracy with which 
known spectral signatures can be used to classify features in 
remotely sensed data acquired from areas other than from which 
the signatures were developed. 

soatial resolution — the smallest size of a uniformly gray object 
that can be recorded by a given sensor on an image that can be 
distinguished from a uniformly gray background at a given con- 
trast ratio. 
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spectral band - discrete segment of the electromagnetic spectrum 
that a sensor is capable of recording. 

threshold - mathematical barrier beyond which a data point (pixel) 
will be excluded from a given class of features. The threshold 
values are controlled by the characteristics of the automated 
classification program employed. 


Vt A-JSC 
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